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ABSTRACT 


The  work  during  this  reporting  period  has  concentrated  on  preparations 
for  a full-scale  test  of  the  ARPA  Authoring  System  in  FY  78.  A five- 
stage  procedure  has  been  developed  for  management  of  task-oriented 
lesson  development  in  the  on-the-job  environment.  It  is  consistent  with 
the  Interservice  Procedures  for  Instructional  System  Development.  It 
is  readily  adapted  to  other  delivery  systems  with  capabilities  similar  to 
the  one  of  choice  here,  the  Lincoln  Terminal  System,  Model  5 (LTS-5). 
A test  of  evaluation  procedures  on  task-oriented  materials  prepared 
under  the  Authoring  System  is  also  reported.  The  plan  to  test  the  system, 
including  preparation  and  evaluation  of  materials  for  both  maintenance 
technicians  avid  technical  operators,  is  reviewed. 
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ARPA  AUTHORING  SYSTEM 


I.  INTRODUCTION 

The  goal  of  the  project  is  to  provide  authoring  procedures  that  enable  subject  matter 
experts  to  prepare  lesson  materials  for  use  in  on-the-job  training.  The  following  conditions 
prevail: 

• A subject  matter  expert,  as  opposed  to  an  instructional  expert, 
is  the  primary  author  of  the  material. 

• Substantial  training  and  guidance  is  given  to  authors  on  computer 
facilities. 

• Materials  are  designed  for  delivery  by  a computer-based  system. 

The  lesson  preparation  procedures  are  consistent  with  the  course  design  principles  underlying 
the  Interservice  Procedures  for  Instructional  Systems  Development.”^ 

The  area  of  application  chosen  involves  experts  in  technical  operations  who  serve  as  authors 
because  acquisition  of  subject  matter  expertise  through  field  experience  is  essential.  Currently, 
bridging  the  gap  between  formal  knowledge  and  actual  readiness  to  perform  complex  tasks  is 
achieved  by  a master/apprentice  training  relationship.  It  is  a costly  method  and  the  shortage 
of  qualified  trainers  is  often  a limiting  factor.  There  is  a need  to  provide  a substitute  for  live 
training.  The  solution  proposed  is  to  use  new  authoring  procedures  developed  especially  for 
this  application  area  to  capture  the  know-how  of  the  technical  expert  in  materials  for  timely 
delivery  by  machine. 

The  approach,  described  in  the  previous  Semiannual  Technical  Summary,^  involves  pro- 
cedures for  authors  that  are  based  on  a prototypical  lesson  design,  one  that  mimics  the  master 
instructing  the  apprentice.  Because  the  target  population  for  task-specific  training  is  often 
small,  authoring  must  proceed  smoothly  and  rapidly  to  justify  the  expenditure  of  technical  talent. 
The  author  must  have  knowledge  of  the  procedures,  operational  requirements,  principles,  facts, 
etc.,  directly  useful  in  task  performance.  A modest  capacity  for  exposition  and  experience  in 
one-on-one  training  are  helpful,  but  the  procedural  approach  to  authoring  greatly  reduces  the 
need  for  the  expert  in  the  subject  matter  to  qualify  as  an  expert  in  educational  strategy  and 
technique  as  well. 

In  order  to  employ  the  authoring  procedure,  it  is  necessary  to  embed  it  in  a training  devel- 
opment system  appropriate  to  the  context  and  goals  of  on-the-job  environments.  One  difficulty 
is  that  the  usual  procedures  are  tailored  to  the  resources  and  management  organization  of  cen- 
tralized, dedicated  training  facilities;  at  the  work  site,  training  is  part  of  the  work  management 
system  and,  since  qualified  technicians  serve  as  authors,  it  is  appropriate  that  lesson  develop- 
ment be  done  within  the  same  organization.  Such  an  arrangement  is  likely  to  make  training 
more  responsive  to  the  needs  of  work  performance.  Another  difficulty  arises  if  there  is  total 
reliance  on  the  usual  ISD  (Instructional  System  Development)  procedure  which  analyzes  tasks 


♦Interservice  Procedures  for  Instructional  Systems  Development,  TRADOC  Pamphlet  350-30, 
U.  S.  Army  (1  August  1975). 

t ARPA  Authoring  System  Semiannual  Technical  Summary  Report,  Lincoln  Laboratory,  M.LT. 
(31  March  1977),  DDC  AD-A044794. 
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TABLE  I 


SUMMARY  OF  THE  STAGES  OF  LESSON  DEVELOPMENT 
WITH  ESTIMATES  OF  DURATION  AND  MAN-HOURS 
(Based  on  Preparation  of  Moterials  to  Support  One  Hour  of  Training) 


STAGE  1 - TRAINING  PROGRAM  PUNNING 

Location 

Work  Site 

Product 

List  of  Lesson  Topics 

Duration 

1 to  2 (doys) 

Personnel 

Work  Manager 

I (man-hours) 

Training  Manager 

1 

STAGE  2 - LESSON  SPECIFICATION 

Locotion 

Work  Site 

Product 

Lesson  Specification 

Lesson  Validation  Plan 

Duration 

2 (days) 

Personnel 

Author 

4 (man-hours) 

Technical  Advisors  (2) 

8 

LTS  Advisor 

4 

Training  Manager 

4 

STAGE  3 - LESSON  PREPARATION 

Location 

Authoring  Center 

Product 

Lesson  in  Draft  Form 

Duration 

30  (days) 

Personnel 

Author 

90  (man-hours) 

LTS  Advisor 

15 

Technical  Advisor 

4 

Trainees 

3 

STAGE  4 - CONVERSION  TO  MEDIUM 

! 

Location 

Authoring  Center,  Fiche  Production  Facility 

Product 

Lesson  on  Microfiche 

Duration 

30  (days) 

Personnel 

Author 

4 (man-hours) 

Artist/Typist 

32 

LTS  Technician 

8 

Photo  Technician 

20 

STAGE  5 - LESSON  EVALUATION 

Location 

Work  Site 

Product 

Lesson  Revisions 

Duration 

10  (days) 

Personnel 

Author 

8 (man-hours) 

LTS  Advisor 

8 

Technical  Advisors  (2) 

8 

into  component  skills  for  subsequent  training.  Instruction  on  component  skills  is  a necessary 
but  not  sufficient  condition  for  preparation  of  the  technician  or  operator  for  work.  It  is  not  a 
substitute  for  hands-on  training  because  many  details  and  the  relevance  of  basic  facts  and  C'n- 
cepts  are  not  adequately  covered.  In  task-oriented  training,  the  relevant  skill  knowledge, 
however  fragmented,  is  presented  at  each  step  of  the  task.  Training  in  general  areas  of  skill 
knowledge  is  presumed  and  the  presentation  is  in  the  form  of  a review  of  related  technical  mat- 
ters. As  a consequence  of  these  differences,  the  ISD  procedures  must  be  adapted  to  the  pro- 
duction of  task-oriented  training  materials. 

A set  of  appropriate  management  procedures  is  presented  in  the  next  section  that  includes 
the  authoring  procedures  previously  developed.  In  the  following  section,  a report  of  a formal 
evaluation  of  a lesson  developed  under  the  authoring  procedures  is  made;  it  constitutes  a dem- 
onstration of  a methodology  for  Stage  5 of  the  materials  development  system.  Evaluation  of 
Training  Effectiveness.  Finally,  a test  plan  is  outlined  for  a full-scale  evaluation  of  the  system 
in  FY  78, 

II.  MANAGEMENT  PROCEDURES  FOR  TASK  LESSON  DEVELOPMENT 

A.  Overview 

A five-stage  procedure  has  been  prepared  for  the  development  of  task  lessons  in  the  on-the- 
job  environment.  The  system  of  materials  preparation  is  tailored  to  meet  three  objectives: 
first,  to  provide  the  work  manager  and  his  technical  staff  with  control  of  the  materials  prepara- 
tion effort;  second,  to  make  efficient  use  of  qualified  experts  in  lesson  preparation;  and,  third, 
to  streamline  the  authoring  process  so  that  the  turnaround  time  for  a lesson  product  is  short. 

The  result  of  each  stage  is  a specific  intermediate  product  along  the  path  of  lesson  devel- 
opment. The  five  stages  are; 

(1)  Planning  a training  program. 

(2)  Preparation  of  a lesson  specification. 

(3)  Preparation  of  a lesson  in  draft  form. 

(4)  Conversion  of  the  materials  to  appropriate  medium. 

(5)  Evaluation  of  training  effectiveness. 

Two  kinds  of  records  are  maintained  on  a routine  basis;  one  indicates  the  efficiency  of  the 
lesson  development  process  and  the  other  the  effectiveness  of  the  final  product.  Data  are  gath- 
ered at  each  stage  to  bring  to  the  attention  of  the  training  system  manager  any  source  of  diffi- 
culty or  delay  in  authoring.  The  information  includes  total  man-hours,  total  calendar  days  to 
completion,  and  quality  of  the  product  at  each  stage.  Effectiveness  is  evaluated  at  Stage  5 on 
the  basis  of  student  records  gathered  during  training.  Eventually,  the  work  manager  executes 
the  lesson  validation  plan  prepared  in  Stage  2 to  determine  whether  lessons  have  the  desired 
effect  on  work  performance. 

The  stages  are  reviewed  in  Table  I;  it  includes  the  locale,  product,  duration,  and  manning 
for  a typical  case.  It  is  assumed  here  that  the  medium  for  training  is  the  LTS-5,  but  the  pro- 
cedures apply  to  any  other  medium  with  similar  capabilities. 


♦"The  LTS-5,  A Brief  Description,"  Technical  Note  1977-13,  Lincoln  Laboratory,  M.I.T. 
(16  December  1977). 
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B.  Stage  1 — Training  Program  Planning 

The  product  of  this  stage  is  a description  of  an  area  of  work  performance  that  is  the  topic 
of  one  or  more  lessons.  Steps  at  this  stage  are: 

(1)  Select  topic. 

(2)  Set  lesson  objectives. 

(3)  Define  the  instructional  setting. 

The  assumption  is  made  that  the  description  of  the  tasks  that  make  up  the  job  have  already  been 
established  according  to  ISD  methodology,  e.g..  Phase  I of  the  Interservice  Procedures.  At  this 
point  the  team  selects  those  topics  related  to  important  work  problems  that  are  appropriate  for 
task-oriented  training. 

C.  Stage  2 - Lesson  Specification 

The  products  of  this  stage  are  a lesson  specification  and  a plan  for  lesson  validation.  The 
procedure  is: 

(1)  Prepare  a lesson  validation  plan. 

(2)  Form  a work  team  of  subject  matter  experts. 

(3)  Identify  the  significant  work  problems. 

(4)  Write  the  lesson  objectives. 

(5)  Identify  typical  work  sequence(s). 

(6)  Review  the  lesson  specification. 

(7)  Select  a team  member  as  author. 

Validation  of  a lesson  requires  monitoring  of  work  performance  under  controlled  conditions. 
Evidence  of  the  quality  of  performance  is  based  on  time  to  complete  tasks,  quality  of  result, 
assessment  of  the  possibility  of  equipment  damage,  and  the  amount  of  supervision  and  assistance 
required.  An  important  aspect  of  the  validation  is  to  check  performance  on  closely  related 
tasks.  These  checks  are  particularly  relevant  to  task  lessons  since  the  purpose  of  including 
explanations  is  to  promote  generalization  to  various  working  conditions. 

The  process  of  lesson  specification  is  to  generate  the  information  listed  in  Table  11.  The 
team  includes  the  author  candidate  and  two  or  three  other  subject  matter  experts.  Operational 
personnel  know  the  work  requirements  but  cannot  be  expected  to  be  aware  of  all  aspects  of  lesson 
planning  and  validation;  thus,  an  educational  specialist,  in  this  instance  the  "LTS  Advisor,"  is 
adso  present  to  assist  in  matters  of  lesson  design  and  planning.  The  author  is  a member  of  the 
team  and  thus  is  prepared  to  deal  with  minor  oversights  in  the  lesson  specification  in  later 
stages.  In  fact,  the  quality  of  the  product  is  judged  in  part  by  the  incidence  of  referrals  back 
to  the  team  for  clarification. 

D.  Stage  3 — Lesson  Preparation 

The  major  product  of  this  stage  is  a draft  lesson  in  notebook  form.  The  notebook  contains 
all  the  basic  information  in  the  lesson,  explanations  and  procedures,  as  well  as  provision  for 
means  to  test  understanding  and  task  performance.  For  LTS-5  delivery,  this  means  rough 
visuals,  an  audio  tape  segmented  by  frames,  and  interactive  frame  logic.  The  notebook  version 
of  the  lesson  is  reviewed  by  the  other  subject  matter  experts  on  the  team,  it  is  tried  out  on  a 
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TABLE  II 

ENTRIES  IN  THE  TASK  LESSON  SPECIFICATION  FORM 


1. 

Course  Title 

4. 

Medium 

2. 

Topic  Area 

5. 

Objectives 

3. 

Training  Conditions 

6. 

Work  Problems 

Trainee 

Prerequisites 

7. 

Typical  Work  Sequence(s) 

Location 

Mode  (hands-on,  simulated) 

8. 

Related  Documents 

Aids 

Trainer 

9. 

Lesson  Title(s) 

Equipment  i 

Documents 

10. 

Validation  Plan 

few  trainees,  and  it  is  revised  if  necessary.  It  remains  unchanged  thereafter  until  it  has  been 
evaluated  in  the  training  environment. 

The  procedure  for  this  stage  is: 

(1)  Prepare  outline  and  flowchart. 

(2)  Prepare  frames. 

(3)  Team  review. 

(4)  Tryout  with  trainees. 

(5)  Notebook  revision. 

The  first  two  steps  constitute  the  task  lesson  [formerly  TPI  (Task  Procedure  and  Interpreta- 
tion)) authoring  procedure.  The  outline  defines  the  steps  in  the  task  and  the  flowchart  represents 
the  organization  of  the  frames.  The  team  reviews  the  notebook,  judging  whether  the  content  is 
adequate  to  meet  the  lesson  objectives  set  down.  Two  or  three  trainees  drawn  from  the  target 
population  are  used  in  the  lesson  trials.  The  author  administers  the  lesson  one-on-one,  simu- 
lating the  functions  normally  performed  by  the  LTS-5  or  other  delivery  system. 

The  lead  roles  throughout  this  stage  are  played  by  the  lesson  author  and  the  LTS  Advisor. 
While  the  identity  of  the  individuals  in  other  roles  may  change  in  the  course  of  lesson  develop- 
ment, there  must  be  a single,  responsible  author.  Any  other  arrangement  is  likely  to  result  in 
an  excessive  rate  of  modification  of  the  lesson,  leading  to  prohibitive  delays  and  costs.  Also, 
to  be  efficient,  the  author's  effort  must  be  a continuous  one,  largely  free  of  interruptions.  The 
LTS  Advisor  is  on  call  during  this  stage  to  answer  the  author's  questions  and  to  comment  on 
design  problems,  strategies,  objectives,  etc.  The  Advisor  resolves  any  issue  concerning  media, 
the  authoring  process,  and  the  lesson  structure.  As  an  expert  on  the  form  of  lessons  but  not 
usually  on  the  subject  matter,  the  Advisor  does  not  normally  comment  on  the  content.  The  au- 
thor is  the  final  judge  of  what  revisions  are  to  be  made  as  a result  of  the  peer  review  and  the 
lesson  trials.  While  all  information  becomes  part  of  the  record  - it  may  be  of  value  for  later 
revisions  — it  is  the  author,  assisted  by  the  other  experts  and  the  LTS  Advisor,  who  prevails. 
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E.  Stage  4 - Conversion  to  Medium 

The  product  of  this  stage  is  a set  of  finished  materials  for  LTS-5  delivery.  The  procedure 
is: 

(1)  Prepare  graphic  displays. 

(2)  Record  audio. 

(3)  Prepare  and  check  frame  logic. 

(4)  Send  materials  to  fiche  production  facility  for  conversion. 

Materials  conversion  must  run  smoothly  to  avoid  turnaround  delays.  The  steps  in  the  overall 
procedure  unique  to  the  LTS  mode  of  delivery  are  mostly  in  this  stage. 

F.  Stage  5 — Lesson  Evaluation 

The  product  of  this  stage  is  a task  lesson  ready  as  a substitute  for  one-on-one  live  instruc- 
tion in  on-the-job  training.  Some  errors  in  form  or  content  may  be  found  in  the  first  version 
and  some  revision  may  be  necessary.  Under  normal  circumstances  a good  result  may  be  ex- 
pected, as  long  as  the  task  procedure  and  the  lesson  in  draft  form  had  previously  been  tried  on 
a few  trainees. 

The  procedure  is: 

(1)  Check  finished  material. 

(2)  Run  lesson  in  training  on  a trial  basis. 

(3)  Collect  and  analyze  training  data. 

(4)  Revise  lesson  as  required. 

The  author  first  checks  the  final  copy  to  be  certain  that  the  lesson  is  free  of  technical  er- 
rors. The  material  is  used  in  the  training  program  using  the  normal  training  procedures,  and 
a record  is  kept  of  the  amount  and  kind  of  assistance  given  by  the  work  supervisor.  Comments 
by  the  trainee,  tests  before  and  after  training,  and  other  information  may  also  be  obtained. 

Lesson  evaluation  is  based  on  the  dwell  time  on  information  frames  and  error  rates  on  the 
quizzes  contained  within  the  lesson.  On  the  LTS-5  these  data  are  recorded  on  cassette  tapes 
that  are  analyzed  subsequently  on  a small,  general-purpose  computer.  These  data  indicate 
areas  of  the  lesson  that  may  require  improvement.  Thus,  short  dwell  times  may  show  the  need 
to  combine  frames  and  long  ones  to  add  new  ones.  Excessive  use  of  Help  and  corrective  frames 
suggests  a need  to  incorporate  more  information  in  the  main  line  of  the  lesson  or  in  the  pre- 
requisite training.  In  general,  objective  evidence  should  carry  much  more  weight  in  the  deci- 
sion to  modify  lessons  than  peer  review,  a rule  that  minimizes  polishing  of  lessons  that  has 
little  return  in  terms  of  training  effectiveness. 

G.  Further  Evaluation 

As  a follow-on,  the  training  manager  will  execute  the  lesson  validation  plan  made  in  Stage  2 
and  gather  evidence  on  the  extent  to  which  training  improves  work  performance.  The  assess- 
ment of  these  results  by  the  work  managers  is,  of  course,  the  ultimate  criterion. 

Records  of  the  man-hours  of  the  author,  LTS  Advisor,  and  other  participants  and  of  the 
elapsed  time  for  each  stage  are  maintained.  These  data  may  be  compared  with  standards  of 
lesson  development  efficiency,  such  as  suggested  in  Table  I,  to  identify  areas  of  difficulty. 
Factors  that  will  affect  the  numbers  include  the  experience  of  the  author  in  preparing  materials, 
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expertise  of  the  author  in  the  topic  area,  and  the  need  for  extensive  explanation  of  procedure  in 
the  lesson.  Provision  of  a system  for  close  monitoring  of  the  lesson  production  process  is  con- 
sistent with  the  ISD  requirements  and  philosophy. 

111.  AN  EXAMPLE  OF  TASK  LESSON  EVALUATION 

An  example  of  an  evaluation  of  a task  lesson,  as  specified  above  in  Stage  5 of  the  lesson 
development  procedures,  is  presented  here. 

A.  Description  of  the  Lesson 

A formal  test  has  been  conducted  of  a task  lesson  reported  in  the  previous  Semiannual 
Technical  Summary  Report.  It  was  described  there  in  Table  1 as  a TPI  under  the  brief  title 
"Voice  Circuits  of  AN/TRC-97A."  The  lesson  was  authored  by  a Senior  Airman  from  the  5th 
Combat  Communications  Group  at  Robins  AFB,  Georgia,  with  the  procedures  developed  under 
the  contract. 

The  subject  matter  is  the  "initial  setup  and  alignment  of  the  AN/TRC-97A  Radio  Set  multi- 
plexer and  installation  of  the  order  wire."  There  are  45  frames  in  all  of  various  kinds.  Nine 
are  introductory.  Nineteen  are  concerned  with  explanations:  6 main  line,  3 Help  (auxiliary 
information),  5 test,  and  5 remedial  frames.  Seventeen  are  task  related:  9 main  line,  4 Help 
(detailed  instructions),  and  4 task  evaluation  frames. 

B.  Test  Procedures 

Several  tests  were  designed  to  evaluate  the  effect  of  the  lesson  on  skill  knowledge  and  work 
performance.  The  Prerequisite  Test  consisted  of  19  multiple-choice  items  to  test  basic  knowl- 
edge of  multiplexers,  receivers,  transmitters,  and  the  concepts  of  signal  and  noise.  The  test 
was  made  up  of  "easy"  items  because  only  an  elementary  knowledge  is  required.  The  Skill 
Knowledge  consisted  of  27  items  on  facts  and  concepts  that  might  be  acquired  during  the  course 
of  the  lesson.  The  Performance  Test  was  made  up  of  two  problems;  each  problem  was  solved 
in  a number  of  steps  and  lead  to  the  identification  and  correction  of  a system  fault.  The  perfor- 
mance was  not  hands-on  to  the  AN/TRC-97A,  and  the  tasks  were  simulated  on  the  LTS-5  instruc- 
tional unit.  An  eight-item  attitude  survey  was  also  given. 

The  test  of  the  lesson  was  preceded  by  familiarization  training.  This  training  on  the 
AN/TRC-97A  consisted  of  rehearsing  some  of  the  same  routine  tasks  done  in  the  test  lesson 
but  without  explanation  of  relevant  concepts.  Approximately  one  week  later  the  main  test  was 
conducted.  First,  the  trainee  was  administered  one  form  each  of  the  Skill  Knowledge  and  the 
Performance  Tests.  The  lesson  was  presented  next,  followed  by  the  other  form  of  the  Skill 
Knowledge  and  Performance  Tests  and  by  the  survey.  The  lesson  was  hands-on  to  the  Radio 
Set.  To  exercise  the  task  instructions,  the  equipment  was  configured  to  simulate  field  conditions 
and  a few  misalignments  were  set  into  the  machine  ahead  of  time. 

The  trainees  consisted  of  both  recent  technical  school  graduates  and  experienced  personnel 
not  familiar  with  the  AN/TRC-97A.  There  were  25  in  all  from  the  2nd  and  5th  Combat  Commu- 
nications Groups  at  Patrick  and  Robins  Air  Force  Bases. 

C.  Results 

The  median  scores  on  the  Prerequisite  Test  were  about  70  percent  for  novice  technicians 
and  90  percent  for  experienced  ones.  The  lowest  score  was  60  percent  in  both  groups.  These 
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data  suggest  that  there  is  substantial  learning  of  basic  concepts  in  the  job  environment.  The 
results  of  the  pre-  and  post-test  of  Skill  Knowledge  show  a median  gain  of  about  40  percent  over 
all  trainees,  an  illustration  of  how  performance  training  enhances  skill  knowledge  learning,  of 
both  procedures  and  facts  and  concepts  in  the  explanations  that  accompany  the  task  instructions. 

The  purpose  of  the  performance  test  was  to  assess  the  impact  of  the  lesson  on  execution  of 
closely  related  tasks,  troubleshooting  in  this  case.  No  gain  or  loss  was  observed  on  the  average. 
The  general  conclusion  of  the  test  results  is  that  there  appears  to  be  learning  of  skill  knowledge 
in  the  job  environment,  but  the  learning  here  was  not  sufficient  to  promote  generalization  to  the 
other  tasks. 

The  LTS  Advisor  reviewed  the  data  obtained  on  the  LTS  cassette  tapes  during  training  and 
identified  one  or  two  places  where  the  explanation  of  important  concepts  was  accompanied  by 
long  frame  dwell  times  and  by  errors  on  the  test  frames  that  followed.  As  a result,  it  was 
recommended  that  a few  frames  be  revised  and  that  the  concept  of  the  system  block  diagram  be 
covered  in  training  prerequisite  to  this  lesson. 

IV.  FIELD  TEST  AND  EVALUATION  OF  THE  AUTHORING  SYSTEM  IN  FY  78 

A full-scale  test  of  the  authoring  system  is  planned  for  FY  78.  The  effort  is  part  of  a 
training  improvement  program  within  the  Tactical  Communications  Area  of  the  Air  Force  Com- 
munications Service.  It  involves  subject  matter  experts  as  authors  from  the  2nd  and  5th  Combat 
Communications  Groups,  Air  National  Guard,  and  Area  headquarters.  The  principal  application 
is  in  the  training  of  Radio  Technicians,  the  304X0  Air  Force  Specialty  Code,  on  maintenance  of 
the  AN/TRC-97A  Radio  Set.  A follow-on  effort  will  be  conducted  in  training  Radio  Operators, 
the  307X0  Code,  on  operation  of  the  AN/TSC-62  Communications  Van. 

The  authors  will  prepare  training  materials  working  within  the  system.  The  lessons  will 
be  incorporated  in  the  regular  program  of  training  and  evaluated.  Thus,  the  effectiveness  of 
the  authoring  system  will  be  assessed  in  terms  of  the  quality  of  the  products.  Formative  eval- 
uation of  lessons  will  be  done  using  the  data  from  10  to  12  trainees  collected  on  the  LTS-5  cas- 
sette tape.  Stage  2 of  lesson  development  includes  specification  of  a lesson  validation  plan;  an 
attempt  will  be  made  to  have  this  plan  executed  for  each  lesson  and  the  results  interpreted. 

It  will  be  recalled  that  the  management  procedures  include  monitoring  the  efficiency  of  the 
lesson  production  process.  Two  measures  are  of  primary  significance,  man-hours  and  elapsed 
time.  A record  of  man-hours,  broken  down  by  work  roles,  provides  a basis  to  estimate  the 
cost  of  lesson  development  in  terms  of  dollars  or,  more  importantly,  in  terms  of  demand  for 
scarce  resources.  A record  of  elapsed  time,  calendar  days  for  each  stage  plus  the  delays  be- 
tween, is  a measure  of  the  responsiveness  of  training  to  the  need  to  correct  work  performance 
problems. 

In  the  field  test,  there  will  be  three  major  parameters:  lesson  complexity,  lesson  prepara- 
tion experience  of  the  author,  and  subject  matter  expertise  of  the  author.  Complexity  of  a task 
lesson  is  defined  in  this  context  as  the  amount  and  sophistication  of  the  explanatory  material 
that  accompanies  the  task  instructions.  At  one  extreme  is  the  pure  procedure,  no  more  than  a 
Job  Performance  Aid  with  task  evaluations  interspersed;  at  the  other  is  the  task  lesson  with 
extensive  interpretation.  The  second  parameter,  authoring  experience,  makes  a distinction 
between  the  subject  matter  expert's  first  and  subsequent  efforts.  Finally,  level  of  expertise 
refers  to  whether  the  author  is  regularly  engaged  in  exercise  of  the  tasks  or  is  familiar  with 
them  only  on  the  basis  of  past  experience.  The  measures  of  the  effects  of  these  parameters 
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will  be  in  terms  of  data  such  as  that  in  Table  I,  man-hours  and  elapsed  time  for  each  Stage. 
Favorable  conditions  are  assumed,  i.e.,  the  task  procedures  are  well  established,  the  lesson 
is  moderate  in  complexity,  and  the  authors  are  qualified  experts  and  have  written  at  least  one 
lesson  before.  The  figures  in  Table  11  will  be  refined  as  a result  of  the  field  tests. 

The  primary  field  test  will  be  conducted  on  lessons  on  the  AN/TRC-9VA  and  authoring 
operations  will  be  centered  at  the  5th  Combat  Communications  Group.  Thirty-two  lessons, 
averaging  | hr  each,  will  be  prepared.  Lessons  will  fall  into  three  categories:  Preventative 
Maintenance  Checks,  Equipment  Alignments,  and  Operations.  The  last  category  is  a miscella- 
neous one,  which  involves  subjects  such  as  Interface  Criteria,  Aligning  Antennas  with  a Pocket 
Transit,  Anti-Jamming  Procedures,  etc.  Some  lessons  planned  are  largely  procedural;  most 
depend  on  extensive  explanations.  Sixteen  authors  will  each  prepare  two  lessons.  Most  are 
actively  engaged  in  doing  the  prescribed  tasks;  others  are  technical  support  personnel.  Thus, 
a mix  of  conditions  will  prevail  that  will  permit  an  assessment  of  the  effects  of  the  main  vari- 
ables — complexity,  authoring  experience,  and  level  of  expertise. 

The  authors  are  given  substantial  administrative  and  technical  support.  They  are  held  ac- 
countable for  their  work  by  senior  members  of  the  authoring  team  who  are  experienced  trainers 
and  are  not  likely  to  permit  lessons  with  serious  flaws  to  be  committed  to  final  form.  As  a 
result,  it  is  anticipated  that  adverse  effects  will  appear  in  the  management  data  rather  than  the 
training  data  - specifically  in  man-hours  and  duration  of  Stage  3,  Lesson  Preparation  itself. 

The  other  area  of  lesson  development,  operation  of  the  AN/TSC-62  Van,  is  of  special  inter- 
est because  of  the  emphasis  on  the  operator  as  trainee,  as  contrasted  to  the  technician.  There 
is  a shift  of  emphasis  toward  memorization  of  complex  procedures  eind  away  from  problem  solv- 
ing that  is  characteristic  of  maintenance  work.  That  is.  understanding  the  tasks  helps  the  op- 
erator to  remember  the  procedures  and  permits  checking  each  step  to  determine  if  it  "makes 
sense."  To  the  technician,  understanding  means  having  facts  amd  concepts  available  that  enter 
into  a reasoning  process  that  leads  to  the  detection  and  correction  of  system  faults.  Presently, 
there  is  a need  to  modify  the  authoring  procedures  in  a few  places  to  accommodate  the  require- 
ments for  operator  training.  The  activity  in  the  area  will  be  centered  at  the  2nd  Combat  Com- 
munications Group  and  will  serve  to  broaden  the  test  of  the  utility  of  the  task  lesson  development 
procedures. 
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